Repository navigation
fix(audit): detect the dcgm-exporter Service on Slurm on Kubernetes - #35
Open
robTheBuildr wants to merge 1 commit into
Open
robTheBuildr wants to merge 1 commit into
robTheBuildr wants to merge 1 commit into
Conversation
On Slurm on Kubernetes the head node is a pod. The GPU Operator runs dcgm-exporter as a DaemonSet in other pods on the GPU nodes, so the head node has no listener on :9400, no systemd unit and no process. The audit reported "DCGM exporter not installed" on a cluster with 18 of 18 exporter pods ready. When the local checks find nothing and resolv.conf has cluster search domains, query the GPU Operator exporter Service and count it only when it serves DCGM_FI_ metrics. The check reads resolv.conf because `su -` login shells drop KUBERNETES_SERVICE_HOST. CLUSTERMAX_DCGM_EXPORTER_URL sets the URL for other namespaces or exporters. Bare-metal head nodes do not query.
robTheBuildr
requested review from
JordanNanos,
Prathmesh234,
functionstackx and
samharshe
as code owners
October 8, 2026 16:40
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #33.
Changes
/etc/resolv.confhas Kubernetes cluster search domains, the Slurm collector queries the GPU Operator exporter Service:http://nvidia-dcgm-exporter.gpu-operator.svc:9400/metrics. The exporter counts only if the response hasDCGM_FI_metrics. The query has a 5 s time limit.KUBERNETES_SERVICE_HOST. The audit can run in asu -login shell, and that shell drops the variable.CLUSTERMAX_DCGM_EXPORTER_URLsets a different URL, for another namespace or exporter. When it is set, the collector queries that URL with no resolv.conf check.CLUSTERMAX_RESOLV_CONFsets the resolv.conf path. The tests use it so that the host resolv.conf does not change the results.Tests
tests/audit/test_monitoring_process_detection.py, 5 new tests:master.dcgm-exporter: detected (DCGM metrics at http://nvidia-dcgm-exporter.gpu-operator.svc:9400/metrics).git diff --checkpassed.Limits
The check shows that the Service serves DCGM metrics. It does not show that all GPU nodes run an exporter, or that Prometheus scrapes it. The standalone collector does not change.
Note
Low Risk
Audit-only monitoring detection with optional outbound HTTP to a configurable metrics URL; no changes to Slurm job control or cluster configuration.
Overview
Extends Slurm cluster audit monitoring detection so dcgm-exporter is found on Slurm-on-Kubernetes head/login pods where the exporter runs as a GPU Operator DaemonSet behind a Service, not on
:9400or as a local systemd unit.When port/systemd checks miss it, the script may
curla metrics URL (5s timeout) and treats the exporter as present only if the body includesDCGM_FI_lines. On Kubernetes it infers a pod fromresolv.confsearch domains (*.svc.) and defaults tohttp://nvidia-dcgm-exporter.gpu-operator.svc:9400/metrics;CLUSTERMAX_DCGM_EXPORTER_URLforces a custom URL without that check, andCLUSTERMAX_RESOLV_CONFpoints at resolv for tests. Bare-metal resolv skips the HTTP probe. The report line now cites how detection succeeded (port/systemd vs metrics URL).Adds five behavioral tests in
test_monitoring_process_detection.pyfor pod detection, non-DCGM/unreachable responses, no curl on host resolv, and URL override.Reviewed by Cursor Bugbot for commit 267bb62. Bugbot is set up for automated code reviews on this repo. Configure here.